• Go to this page: http://bit.ly/bigdatastat !
• Fill in some keywords related to Big Data!

• Ask 40 experts to define "big data"…

• Ask 40 experts to define "big data"…
• … get 40 different definitions :)

"Big Data is the result of collecting information at its most granular level — it’s what you get when you instrument a system and keep all of the data that your instrumentation is able to gather."

Jon Bruner (Editor-at-Large, O’Reilly Media)

"Big data is data that contains enough observations to demand unusual handling because of its sheer size, though what is unusual changes over time and varies from one discipline to another."

Annette Greiner
(Lecturer, UC Berkeley School of Information)

"[…] 'big data' will ultimately describe any dataset large enough to necessitate high-level programming skill and statistically defensible methodologies in order to transform the data asset into something of value."

Reid Bryant
(Data Scientist, Brooks Bell)

• Large amounts of data
• Various types/formats of data
• Unusual sources
• Speed of data flow/stream
• Use programming and statistics (in a broad sense) to extract value

• How to design/set-up the machinery to handle large amounts of data?
• How to use the existing machinery most efficiently for large amounts of data?
• How to approach the analysis of large amounts of data with statistics?

• How to design/set-up the machinery to handle large amounts of data?
• How to use the existing machinery most efficiently for large amounts of data?
• How to approach the analysis of large amounts of data with statistics?

• How to design/set-up the machinery to handle large amounts of data?
• How to use the existing machinery most efficiently for large amounts of data?
• How to approach the analysis of large amounts of data with statistics?
  1. Compute 'usual' statistics based on large dataset (many observations).

• How to design/set-up the machinery to handle large amounts of data?
• How to use the existing machinery most efficiently for large amounts of data?
• How to approach the analysis of large amounts of data with statistics?
  1. Compute 'usual' statistics based on large dataset (many observations).
  2. Multivariate Statistics to gain insights from Big Data (many variables).

• Mother nature always has provided the data, but…
  • … instruments have gotten more precise
  • … new measurement methods have been developed
• Prominent examples: Astronomy, Genomics/Bioinformatics

Photo by Joe Parks, (CC BY-NC 2.0) source: https://flic.kr/p/e2umhv

Astronomy: SKA Radio Telescope

Image by SKA Organisation, source: https://www.skatelescope.org/multimedia/image

Astronomy: SKA Radio Telescope

Image by SKY Organisation, source: https://www.skatelescope.org/multimedia/image

• Hardware: Diffusion of the Internet and mobile-phone networks.
• Software: Web 2.0 Technologies (APIs, JSON, Programmable Web, etc.).

• Hardware: Diffusion of the Internet and mobile-phone networks.
• Software: Web 2.0 Technologies (APIs, JSON, Programmable Web, etc.).
  • Backbone of social media and many prominent web services (e.g., Google Maps).
  • Data integration across platforms and services.
  • Exchange of data between/across applications.

• Often tied to web applications and digitization of economic and political processes.

• Often tied to web applications and digitization of economic and political processes.
• Volume of data is substantial (but usually smaller than in the natural sciences).

• Often tied to web applications and digitization of economic and political processes.
• Volume of data is substantial (but usually smaller than in natural sciences)
• Variety and variability often more challenging than in natural sciences.
  • Various sources
  • Data generation/sensors are independent from research endeavor.

• Often tied to web applications and digitization of economic and political processes.
• Volume of data is substantial (but usually smaller than in natural sciences)
• Variety and variability often more challenging than in natural sciences.
  • Various sources
  • Data generation/sensors are independent from research endeavor.
• Questions/Problems often similar to applied research in the industry.
  • Key difference: usually no streaming applications (velocity not that much of an an issue).

1. Big Data: Basic Concepts and Applications in R (Ulrich Matter)
2. Multivariate Statistics in Python (Matthias Fengler)

‐ Understand the concept of Big Data in the context of economic research.

‐ Understand the technical challenges of Big Data Analytics and how to practically deal with them.

‐ Students will know the basic statistical techniques of clustering, dimensionality reduction, and factor models.

• Decentral ‐ Group examination 'paper' (all given the same grades) (50%).
• Group size: 2 to max 3. people.
• Take‐home exercises (group task): Application of basic concepts in R when working with big data. Conceptual questions related to the application.

• Decentral ‐ Oral examination (individual) (50%, 15 mins.)
• Oral exam: Multivariate statistics (methods/concepts).

1. Analyze the underlying problem.
2. Understand the underlying concept.
3. Apply the understanding in a hands-on exercise.

• A tool to analize problems posed by large datasets.
  • For example, memory usage (in R).
• A practical tool for Big Data Analytics.

Preparations

# read dataset into R
economics <- read.csv("../data/economics.csv")
# have a look at the data
head(economics, 2)

##         date   pce    pop psavert uempmed unemploy
## 1 1967-07-01 507.4 198712    12.5     4.5     2944
## 2 1967-08-01 510.5 198911    12.5     4.7     2945

# create a 'large' dataset out of this
for (i in 1:3) {
     economics <- rbind(economics, economics)
}
dim(economics)

## [1] 4592    6

Compute the real personal consumption expenditures (pce): Divide each value of pce by the deflator 1.05.

# Naïve approach (ignorant of R)
deflator <- 1.05 # define deflator
# iterate through each observation
pce_real <- c()
n_obs <- length(economics$pce)
for (i in 1:n_obs) {
  pce_real <- c(pce_real, economics$pce[i]/deflator)
}

# look at the result
head(pce_real, 2)

## [1] 483.2381 486.1905

How long does it take?

# Naïve approach (ignorant of R)
deflator <- 1.05 # define deflator
# iterate through each observation
pce_real <- list()
n_obs <- length(economics$pce)
time_elapsed <-
     system.time(
         for (i in 1:n_obs) {
              pce_real <- c(pce_real, economics$pce[i]/deflator)
})

time_elapsed

##    user  system elapsed 
##   0.163   0.012   0.176

Assuming a linear time algorithm (\(O(n)\)), we need that much time for one additional row of data:

time_per_row <- time_elapsed[3]/n_obs
time_per_row

##      elapsed 
## 3.832753e-05

If we deal with big data, say 100 million rows, that is

# in seconds
(time_per_row*100^4) 

##  elapsed 
## 3832.753

# in minutes
(time_per_row*100^4)/60 

##  elapsed 
## 63.87921

# in hours
(time_per_row*100^4)/60^2 

##  elapsed 
## 1.064654

What happens in the background?

• Evaluation/computation
• Memory allocation/deallocation

Can we improve this?

# Improve memory allocation (still somewhat ignorant of R)
deflator <- 1.05 # define deflator
n_obs <- length(economics$pce)
pce_real <- list()
# allocate memory beforehand
# tell R how long the list will be
length(pce_real) <- n_obs

Can we improve this?

# Improve memory allocation (still somewhat ignorant of R)
deflator <- 1.05 # define deflator
n_obs <- length(economics$pce)
pce_real <- list()
# allocate memory beforehand
# tell R how long the list will be
length(pce_real) <- n_obs
# iterate through each observation
time_elapsed <-
     system.time(
         for (i in 1:n_obs) {
              pce_real[[i]] <- economics$pce[i]/deflator
})

time_elapsed

##    user  system elapsed 
##   0.028   0.001   0.028

Any improvements?

time_per_row <- time_elapsed[3]/n_obs
time_per_row

##      elapsed 
## 6.097561e-06

# in seconds
(time_per_row*100^4) 

##  elapsed 
## 609.7561

# in minutes
(time_per_row*100^4)/60 

## elapsed 
## 10.1626

# in hours
(time_per_row*100^4)/60^2 

##   elapsed 
## 0.1693767

This looks much better, but we can do even better…

Can we improve this?

# Do it 'the R wqy'
deflator <- 1.05 # define deflator
# Exploit R's vectorization!
time_elapsed <- 
     system.time(
     pce_real <- economics$pce/deflator
          )
# same result
head(pce_real, 2)

## [1] 483.2381 486.1905

# but much faster!
time_elapsed

##    user  system elapsed 
##       0       0       0

time_per_row <- time_elapsed[3]/n_obs

In fact, system.time() is not precise enough to capture the time elapsed…

# in seconds
(time_per_row*100^4) 

## elapsed 
##       0

# in minutes
(time_per_row*100^4)/60 

## elapsed 
##       0

# in hours
(time_per_row*100^4)/60^2 

## elapsed 
##       0

1. How R allocates and deallocates memory can have a substantial effect on computation time.
   • (Particularly, if we deal with a large dataset!)
2. In what way the computation is implemented can matter a lot for the time elapsed.
   • (For example, loops vs. vectorization/apply)

Book

Walkowiak, Simon (2016): Big Data Analytics with R.Birmingham, UK: Packt Publishing.

Journal Articles

Wickham, Hadley and Dianne Cook and Heike Hofmann (2015): Visualizing statistical models: Removing the blindfold.Statistical Analysis and Data Mining: The ASA Data Science Journal. 8(4):203‐225.

Schwabish, Jonathan A. (2014): An Economistʹs Guide to Visualizing Data.Journal of Economic Perspectives. 28(1):209‐234.

• umatter.github.io/courses
• github.com/umatter/BigData
• StudyNet

• Clone/fork the course's GitHub-repository
• During class, use the Rmd-file of the slide-set as basis for your notes
• After class, enrich/merge/extend your notes with the lecture notes.

• Install R, RStudio
• Set up your own GitHub-account
• Get familiar with Git/GitHub